Wow, Kevin. That's excellent support, thank you so much for making the effort!
I had no idea that using concurrency in file system calls would have any such effect. You proved me wrong.
In a real implementation, I think you'd probably want a fixed set of threads calling "getattrlistbulk", another set of threads processing the data each call produced, and then system the pushed directory back to the syscall thread.
Already doing all the processing past the initial gathering of the directory contents in a separate thread, but that's clearly far from what's possible.
I actually tracked this bug down and I have a question about your testing. Were you always testing on "freshly" mounted volumes (meaning, umount volume-> mount volume-> searchfs)? Or did you also test on volumes under normal use and repeatedly running searchfs?
The use case of my app is to mainly run searches on the entire startup volume. So that's what I focused on with my tests.
I try to be aware of caching effects (I believe some OS versions can use a of the available memory for caching information, but never learned about the details - all info on this I found was fairly vague). What I did, for instance, was to run searchfs on a running High Sierra system using HFS+, then cloned the system and let it convert to APFS, booted from that and ran the same tests. Repeatedly, of course. Later ran the tests on upgraded macOS systems, showing no significant improvement.
The thing that confuses me most about searchfs performance on APFS is that when I search for something that's not findable (e.g. a unique name) both with searchfs and with a conventional method that eventually goes thru getatrrlistbulk, it didn't make much of a difference - even though the number of syscalls should be immensely different (and thereby contributing a lot to the overall time): searchfs should be a single syscalls, whereas the other method will be in the 100000s, right? That is why it could be so fast on HFS+, because all the searching happened right in the VFS, in a tight loop.
Am I misunderstanding how it's working on APFS? I always assumed it was because of the dual-tree nature of APFS, where one cannot simply look at the one tree containing the dir records, looking for matching names, but must also look up the corresponding entries in the other tree in order to figure out which volume the dir entries belong to, and this would be the bottleneck (though one could argue that simply first blindly looking for matches in the dir rec tree and only then verifying the other tree if a match is found may be faster than the other way around that I suspect it's happening now). You understand what I mean? searchfs may be optimized to either first sort out the right dir recs by making sure they belong to the target volume before checking the dir properties, or vice versa, and maybe the other way around would be usually faster (assuming one seeks to find a small set of hits). Of course, that would require more work on this code for very few gain (hardly anyone is using this anyway, I believe).
FYI, "find" is opensource and seems to be a relatively straightforward implementation built on fts.
Yes, I know, and still my fts code in DirScanner was slower. So I assumed it had to do with the use of NSURLs.
However, I must now admit that I was wrong in that. While
find is still twice as fast compared to the search performance in FindAnyFile, it is now on-par with my most recent DirScanner version: After Jim Luther told me about the key to using fileSystemRepresentation instead of converting paths to NSStrings in my code, DirScanner became significantly faster. Yet, I failed to contemplate that fact when I wrote above that fts would still be 2x faster, or was too lazy (or confident) to double check.
Now, with your findings on running parallel dir reads, I have a big task ahead of me rewriting that code. I'm excited. Thank you again for all your input.
